The real time fusion of multi-sensor imagery is an important and growing area in image processing. An example of a system incorporating multiple sensors is the ITT Enhanced Night Vision Goggle (hereinafter “ENVG”). The ENVG is a head-borne night vision multi-spectral system which uses optical overlay fusion to fuse digital imagery from a thermal camera with imagery from a conventional direct-view light intensification tube. In the ENVG, the video imagery provided by the thermal sensor is presented digitally through an LCD display device and is overlaid onto the imagery from the intensification tube using a beam combiner.
Many fusion applications require that a sensor pair operate in a dynamic environment where, either the sensors are moving relative to the scene, or objects in the scene are moving relative to the sensors. In any real-time image fusion system, it is likely that a time lag disparity exists between the sensors in the video chains such that the time from input of photons to output of photons (or output of electrons) is not the same for every sensor. In the ENVG, for example, there is a time lag disparity between the output of a direct-view light intensification tube and the output of a thermal camera.
Any attempt to combine the video imagery from multiple sensors results in image mismatch or misregistration caused by the time lag disparity between the multiple input systems. If the disparity in one sensor is large compared to the video frame rate of a faster sensor, a noticeable separation between the multiple video images becomes apparent to the viewer whenever motion is present in the scene. In such a circumstance, one sensor is known to “lag” the other sensor. The amount of image mismatch depends directly on the severity of the sensor lag and the relative velocity between the camera's focal plane and the objects in the scene.
There exists a need for a system and a method for compensating image mismatch among video images provided by multiple video image sensors, in which the mismatch is caused by movement of the sensors and by time lag disparities introduced by the video image sensors.